Hierarchical Clustering

The Presidents are clustered based on the similarity of their State of the Union texts using the Jensen–Shannon divergence method. There are approximately 5 groups, and it is apparent that the presidents of similar eras are grouped together.

K-Means Clustering

The Calinski-Harabasz function suggests that 6 is the optimal number of K-Means clusters to group the presidents.

K-Means clustering also shows that the presidents are grouped together by similar eras based on their PCA features.

Word Associations

The following word associations shows how Democratic and Republican presidents differed in their choice of terms in their State of the Union speeches.

Democratic presidents when they discussed “freedom”:

## $freedom
##     smashing   subjugated       speech     espousal     matching 
##         0.22         0.22         0.20         0.17         0.17 
##     enslaved   liberating        lords     religion   likelihood 
##         0.13         0.13         0.13         0.13         0.12 
## conscription      scratch        vivid    objective        world 
##         0.11         0.11         0.10         0.09         0.09 
##  adversaries    eternally   expression   militarism       serves 
##         0.08         0.08         0.08         0.08         0.08 
##   translated    blessings         foes  ideological        peace 
##         0.08         0.07         0.07         0.07         0.07 
##      reduces         fear        flags independence      undergo 
##         0.07         0.06         0.06         0.06         0.06 
##    wondering 
##         0.06

Republican presidents when they discussed “freedom”:

## $freedom
##      cambodia      fighters   freedomsour        angola        defend 
##          0.17          0.15          0.14          0.10          0.10 
##   indivisible   interlocked  rightfulness        speech         world 
##          0.09          0.09          0.09          0.09          0.09 
##         burma         peace       worship       america      assemble 
##          0.08          0.08          0.08          0.07          0.07 
##       belarus         cause      champion       consign     democracy 
##          0.07          0.07          0.07          0.07          0.07 
##      disagree        faiths         fight          free    imprisoned 
##          0.07          0.07          0.07          0.07          0.07 
## individuality       planted      swelling           usa     wednesday 
##          0.07          0.07          0.07          0.07          0.07 
##      zimbabwe   afghanistan   aspirations    democratic         human 
##          0.07          0.06          0.06          0.06          0.06 
##    individual       proudly     religious          sees     spreading 
##          0.06          0.06          0.06          0.06          0.06 
##     tolerance           win     elections   foundations     frontiers 
##          0.06          0.06          0.05          0.05          0.05 
##          hate       liberty          tide        values 
##          0.05          0.05          0.05          0.05

Democratic presidents when they discussed “budget”:

## $budget
##        balanced     unbalancing            cuts         pleaded 
##            0.26            0.16            0.13            0.12 
##          fiscal           plead            ance             bal 
##            0.11            0.11            0.10            0.10 
##       balancing         deficit          octdec octoberdecember 
##            0.10            0.10            0.10            0.10 
##     pricefixing         balance         billion    expenditures 
##            0.10            0.09            0.09            0.09 
##         federal        includes        antidrug         defense 
##            0.09            0.08            0.07            0.07 
##    entitlements              fy    racketeering        requests 
##            0.07            0.07            0.07            0.07 
##        responds        spending            cash   congressional 
##            0.07            0.07            0.06            0.06 
##         ensures         expands        programs 
##            0.06            0.06            0.06

Republican presidents when they discussed “budget”:

## $budget
##            balanced            ravaging            spending 
##                0.26                0.22                0.14 
##        expansionary           balancing        incorporates 
##                0.12                0.11                0.11 
##        jobproducing          scheduling             spelled 
##                0.11                0.11                0.11 
##           unbalance            whittier              freeze 
##                0.11                0.11                0.10 
##            trillion             billion             deficit 
##                0.10                0.09                0.09 
##              submit            airpower                 epa 
##                0.09                0.08                0.08 
##             federal grammrudmanhollings          priorities 
##                0.08                0.08                0.08 
##                sets             targets           budgetary 
##                0.08                0.08                0.07 
##             current              fiscal         forthcoming 
##                0.07                0.07                0.07 
##           indicates            lineitem            priority 
##                0.07                0.07                0.07 
##             balance          comparable             defense 
##                0.06                0.06                0.06 
##             earmark              funded              modest 
##                0.06                0.06                0.06 
##           spectacle            totaling 
##                0.06                0.06

Democratic presidents when they discussed “energy”:

## $energy
##    decontrolled     exploratory         gasohol        unleaded 
##            0.38            0.38            0.38            0.38 
##        windfall           solar    conservation             gas 
##            0.38            0.32            0.30            0.28 
##         alcohol        gasoline      quadrupled           slope 
##            0.27            0.27            0.27            0.27 
##       synthetic           clean           fuels           crude 
##            0.27            0.26            0.25            0.24 
##       renewable      incentives    dramatically        drilling 
##            0.24            0.20            0.19            0.19 
##       rationing          atomic         gallons      households 
##            0.17            0.16            0.16            0.16 
##         enacted              fy         natural      production 
##            0.15            0.15            0.15            0.15 
##      buttressed      costimpact       currently     deregulated 
##            0.14            0.14            0.14            0.14 
## egyptianisraeli    evenhandedly        helsinki      indigenous 
##            0.14            0.14            0.14            0.14 
## industrializing     objectively        premised         rebates 
##            0.14            0.14            0.14            0.14 
##     redirection   reorientation       lowincome         sources 
##            0.14            0.14            0.13            0.13 
##         funding       increased 
##            0.12            0.12

Republican presidents when they discussed “energy”:

## $energy
## revitalization         atomic        cleaner         floors     technology 
##           0.25           0.24           0.19           0.19           0.16 
##   conservation          solar          clean          atoms         enacts 
##           0.15           0.15           0.14           0.13           0.13 
##     geothermal           grid      petroleum     accelerate      shortages 
##           0.13           0.13           0.12           0.11           0.11 
##      consuming    electricity   independence     allocation   deregulating 
##           0.10           0.10           0.10           0.09           0.09 
##        develop            gas      generates       reliable      stockpile 
##           0.09           0.09           0.09           0.09           0.09 
##           wind  breakthroughs  comprehensive        nuclear 
##           0.09           0.08           0.08           0.08

Democratic presidents when they discussed “security”:

## $security
##        social       israels      medicare beneficiaries    collective 
##          0.35          0.10          0.10          0.08          0.08 
##     crediting       europes         havel          lech          thai 
##          0.08          0.08          0.08          0.08          0.08 
## thaicambodian        vaclav        walesa      medicaid        afghan 
##          0.08          0.08          0.08          0.07          0.06 
##          aged       derives      facility        health        israel 
##          0.06          0.06          0.06          0.06          0.06 
##    livelihood      national     repassing       seniors       council 
##          0.06          0.06          0.06          0.06          0.05 
##     enhancing     guarantee 
##          0.05          0.05

Republican presidents when they discussed “security”:

## $security
##       social     homeland  revitalized          fbi   collective 
##         0.25         0.15         0.14         0.12         0.11 
##   retirement   reinforced      pundits bioterrorism        pacts 
##         0.11         0.10         0.09         0.08         0.08 
##  attachments      council   dedication      doubles     medicaid 
##         0.07         0.07         0.07         0.07         0.07 
##     medicare   practicing   reaffirmed          rob     survivor 
##         0.07         0.07         0.07         0.07         0.07 
## unchallenged   bipartisan    broadened  commitments costofliving 
##         0.07         0.06         0.06         0.06         0.06 
##      defense     diverted  entitlement      focused       funded 
##         0.06         0.06         0.06         0.06         0.06 
##        peace   powerfully     priority           rd    southeast 
##         0.06         0.06         0.06         0.06         0.06 
## strengthened      younger         asia         boom   conserving 
##         0.06         0.06         0.05         0.05         0.05 
##       mutual     national   strategies       vanish 
##         0.05         0.05         0.05         0.05

Democratic presidents when they discussed “economy”:

## $economy
##            global             rigid         lifeblood          deprives 
##              0.13              0.13              0.10              0.09 
##       foreignflag         seafaring           sixpart        straitened 
##              0.09              0.09              0.09              0.09 
##         unbalance            usbulk         worsening       environment 
##              0.09              0.09              0.09              0.07 
##           growing            revive        simplicity balanceofpayments 
##              0.07              0.07              0.07              0.06 
##           barring          combines       competitive        efficiency 
##              0.06              0.06              0.06              0.06 
##         expanding         fashioned             frown              jobs 
##              0.06              0.06              0.06              0.06 
##             rated         recession              bust          depleted 
##              0.06              0.06              0.05              0.05 
##           dynamic             fiber           longrun           nurture 
##              0.05              0.05              0.05              0.05 
##            shrink           stamina         stringent            strong 
##              0.05              0.05              0.05              0.05 
##            talent 
##              0.05

Republican presidents when they discussed “economy”:

## $economy
##        clamored     misdirected parenthetically   mismanagement 
##            0.20            0.20            0.20            0.16 
##       practised         praised      efficiency   quartermaster 
##            0.14            0.14            0.11            0.11 
##       expanding         growing    halftrillion      commissary 
##            0.10            0.10            0.10            0.09 
##            jobs         healthy          strong        denounce 
##            0.09            0.08            0.08            0.07 
## noninflationary     competitive    constructive    expansionary 
##            0.07            0.06            0.06            0.06 
##           grows  industrialized       peacetime       recession 
##            0.06            0.06            0.06            0.06 
##    retrenchment        stronger     underground         wartime 
##            0.06            0.06            0.06            0.06 
##          begins    inflationary      transition 
##            0.05            0.05            0.05

Text Hierachical Clustering

Term hierachical clustering shows how frequent terms appeared together in each president’s speeches and highlights the presidents’ policies that were presented to Congress and to the American people.

Word clouds

The following word clouds highlight the frequent terms used by every president and show how the State of the Union speeches evolved by presidency.

George Washington (1790-1796)

John Adams (1797-1800)

Thomas Jefferson (1801-1808)

James Madison (1809-1816)

James Monroe (1817-1824)

John Quincy Adams (1825-1828)

Andrew Jackson (1829-1836)

Martin van Buren (1837-1840)

John Tyler (1841-1844)

James Polk (1845-1848)

Zachary Taylor (1849)

Millard Fillmore (1850-1852)

Franklin Pierce (1853-1856)

James Buchanan (1857-1860)

Abraham Lincoln (1861-1864)

Andrew Johnson (1865-1868)

Ulysses S. Grant (1869-1876)

Rutherford B. Hayes (1877-1880)

Chester A. Arthur (1881-1884)

Grover Cleveland (1885-1888, 1893-1896)

Benjamin Harrison (1889-1892)

William McKinley (1897-1900)

Theodore Roosevelt (1901-1908)

William H. Taft (1909-1912)

Woodrow Wilson (1913-1920)

Warren Harding (1921-1922)

Calvin Coolidge (1923-1928)

Herbert Hoover (1929-1932)

Franklin D. Roosevelt (1934-1945)

Harry S. Truman (1946-1953)

Dwight D. Eisenhower (1953-1961)

John F. Kennedy (1961-1963)

Lyndon B. Johnson (1964-1969)

Richard Nixon (1970-1974)

Gerald R. Ford (1975-1977)

Jimmy Carter (1978-1981)

Ronald Reagan (1982-1988)

George H.W. Bush (1989-1992)

William J. Clinton (1993-2000)

George W. Bush (2001-2008)

Barack Obama (2009-2015)